Skip to content

Remove external untokenize package. Use tokenize from the Python standard library.#325

Merged
weibullguy merged 1 commit into
PyCQA:masterfrom
tdenewiler:untokenize-std-lib
Oct 29, 2025
Merged

Remove external untokenize package. Use tokenize from the Python standard library.#325
weibullguy merged 1 commit into
PyCQA:masterfrom
tdenewiler:untokenize-std-lib

Conversation

@tdenewiler
Copy link
Copy Markdown
Contributor

No description provided.

@tdenewiler
Copy link
Copy Markdown
Contributor Author

This is required for Python 3.14. The untokenize package used by docformatter has not received any new commits in 6 years.

Tested with

docker run -it python:3.14 bash
cd /tmp/
git clone https://github.com/tdenewiler/docformatter.git
cd docformatter
git checkout untokenize-std-lib
pip install tox
tox -e py314

All tests are passing. If you stay on master branch and do not checkout the new branch then those same tests fail.

@tdenewiler
Copy link
Copy Markdown
Contributor Author

I ran the tox tests on Python 3.10, 3.11, 3.12, 3.13, and 3.14.

@adamtheturtle
Copy link
Copy Markdown
Contributor

The key insight here is that untokenize isn't even used. It is an unnecessary dependency.

FWIW I recommend using deptry to find unused dependencies.

@jonathan343
Copy link
Copy Markdown

+1 for removing this unused dependency

@adamtheturtle
Copy link
Copy Markdown
Contributor

@weibullguy Hopefully this is a very clear "merge", and a release to fix 3.14 would be welcome

@jonathan343
Copy link
Copy Markdown

I'll probably look into the pydocstyle plugin for ruff. It supports a pep257 convention which seems to actually provide more coverage than this repo. Posting incase anyone else is looking for an alternative

@weibullguy
Copy link
Copy Markdown
Member

I'll probably look into the pydocstyle plugin for ruff. It supports a pep257 convention which seems to actually provide more coverage than this repo. Posting incase anyone else is looking for an alternative

Does this plugin format docstrings or just verify them against various standards?

@adamtheturtle
Copy link
Copy Markdown
Contributor

FWIW even with this PR, on 3.14 docformatter adds extraneous newlines.

@weibullguy weibullguy merged commit e239dca into PyCQA:master Oct 29, 2025
7 checks passed
@tdenewiler tdenewiler deleted the untokenize-std-lib branch November 16, 2025 03:27
weiji14 added a commit to regro-cf-autotick-bot/docformatter-feedstock that referenced this pull request Apr 28, 2026
weiji14 added a commit to conda-forge/docformatter-feedstock that referenced this pull request Apr 29, 2026
* updated v1.7.8

* MNT: Re-rendered with conda-smithy 3.61.2 and conda-forge-pinning 2026.04.28.18.11.5

* Convert from v0 meta.yaml to v1 recipe.yaml syntax

* MNT: Re-rendered with conda-smithy 3.61.2 and conda-forge-pinning 2026.04.28.18.11.50

Other tools:
- conda-build 25.11.1
- rattler-build 0.63.1
- rattler-build-conda-compat 1.4.13

* Bump python_min from 3.9 to 3.10

Xref PyCQA/docformatter@6a90c74

* Remove untokenize runtime dependency

Xref PyCQA/docformatter#325

* Extend python_version in tests

as per https://conda-forge.org/docs/maintainer/example_recipes/pure-python/#the-tests-section

---------

Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>
Co-authored-by: Carlo <carlodri@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants